Data center modules and method of large-scale deployment

ABSTRACT

A data center module is a data center that can be prefabricated using generally standardized off-the-shelf components, and quickly assembled on a collocation site where a shared central facility is provided. The data center module is typically configured to be deployed with other identical data center modules around the central facility both in side-to-side and/or in back-to-back juxtapositions, typically without the need for interleaving space between adjacent modules in order to maximize real estate use. Each data center module typically comprises harden party walls, several floors for accommodating all the necessary electrical and cooling subsystems and for accommodating all the computing machinery (e.g. servers). Though all the data center modules share similar physical configuration, each data center module can be independently customized and operated to accommodate different needs. Each data center module also incorporates a highly efficient hybrid cooling system that can benefit from both air-side and water-side economizers.

Cross-Reference to Related Applications

The present patent application is a divisional application of commonlyassigned U.S. patent application Ser. No. 14/577,276 entitled “DataCenter Modules and Method of Large-Scale Deployment” and filed at theUnited States Patent and Trademark Office on Dec. 19, 2014, itself adivisional application of commonly assigned U.S. patent application Ser.No. 13/746,042 entitled “Prefabricated Energy Efficient Data CenterCondominiums and Method of Large Scale Deployment” and filed at theUnited States Patent and Trademark Office on Jan. 21, 2013, itselfclaiming priority of U.S. Provisional Patent Application No. 61/736,270,entitled “Prefabricated Energy Efficient Data Center Condominiums andMethod of Large Scale Deployment” and filed at the United States Patentand Trademark Office on Dec. 12, 2012. The present application claimsthe benefits of priority of all these prior applications. Thedisclosures of these prior applications are incorporated herein byreference.

FIELD OF THE INVENTION

The present invention generally relates to data centers and moreparticularly to modular data centers and data center modules.

BACKGROUND OF THE INVENTION

Modularity, scalability and flexibility are now essential requirementsfor efficient and cost effective data centers. Modularity is thebuilding block that allows rapid on-demand deployment ofinfrastructures. Modularity minimizes capital expenditure and, thus,maximizes return on investment (ROI). Scalability relates to modularity,but is the additional key that enables a design to scale past thebarrier of a predetermined fixed number of modules. It is the glue thatallows the different types of modules to coherently scale: specificallycomputing modules with floor/space modules, power modules, and coolingmodules. Flexibility further refines modularity and scalability byallowing any type of hardware from any vendor, with various power andcooling requirements, to coexist within the same data center. It is mostcrucial in the context of serving multiple independent users choosing tocollocate their specific computing machinery in a shared data center.

Recent power density increases in computer packaging are amongst thegreatest limiting factors of scalability and flexibility in datacenters. Current best practices suggest to partition large computingrooms into low, medium, and high power density zones. In this way, alimited form of scalability and flexibility can be reached, negating theneed to overprovision the whole computing room with the highest possiblepower density capability. Nevertheless, forcing these zones to be sizeda priori is hardly modular. The problem lies with the conventional datacenter design where a huge computing room is surrounded byproportionally sized mechanical and electrical rooms. Such arrangementsare difficult to scale, because large distances limit our ability toefficiently distribute low voltage power to computing machinery, andmove enough air to keep this machinery cool. Air cooling at large scalesespecially becomes daunting, because air velocity needs to be kept atacceptable levels using air conduits with limited cross-sections. Toomuch air velocity brings turbulence that in turn produces pressuredifferentials and, thus, non uniform air distribution and poor coolingefficiency. Moving water over large distances is both much easier andefficient. However, bringing water all the way to the computer cabinet(or even inside the cabinets) creates other challenges like leakdetection and proofing.

Another popular trend is to use shipping containers to hostpreconfigured and preassembled computing hardware. Although thisapproach can be very modular and, to some extent, scalable, it is not somuch flexible. The physical dimensions of a standard shipping containerimpose severe space constraints that usually limit the computer formfactors that can be hosted while rendering hardware maintenanceoperations more difficult. Promoters of this approach are often hardwarevendors of some sort, using the container model to push their ownhardware as the backbone of data centers. Container based data centersare most practical when computing resources need to be mobile for somereason. In practice, however, even though rapid initial deployment is anobvious competitive advantage, rapid redeployment is a rare requirementbecause of the relative short lifespan of computers. Moreover, there isthe additional issue of the low voltage power feeds usually required bythese containers that have limited space for in-container powertransformation. For large scale configurations, this forces either toinefficiently carry low voltage energy over large distances, or tocombine computing containers with power transformation containers.

Finally, energy efficiency is also a very important requirement formodern data centers, both because of its financial and environmentalimpact. The two main sources of power losses in data centers lie involtage transformation and regularization, on the one hand, and heatdisposal, on the second hand. Best practices for efficient electricalsystems are to minimize the number of voltage transformation stages andto transport energy at higher voltage. Also, it is important tocorrectly size the electrical infrastructure according to effectiveneeds, as underutilized electrical systems are usually less efficient.As for efficient heat disposal, there are mostly air-side and water-sideeconomizers to exploit favorable outside climate conditions to totallyor partially circumvent the need for power hungry chillers. The holisticproblem, however, is how to design cost-effective and energy efficientdata centers that are also modular, scalable, and flexible.

In view of the foregoing, there is a need for an improved data centermodule which mitigates at least some shortcomings of prior data centermodules.

SUMMARY OF THE INVENTION

A data center module in accordance with the principles of the presentinvention generally mitigates at least some of the shortcomings of priordata center modules by comprising multiple levels configured toaccommodate both the cooling and the electric subsystems and thecomputing machinery (e.g. servers), and by being configured to bedeployed with other identical data center modules around a centralshared facility.

A data center module in accordance with the principles of the presentinvention generally comprises a compact-footprint weatherproof envelop,complete with party walls and staging areas, and a multistory energyefficient layout capable of powering and cooling typically genericcomputer hardware. The module therefore generally comprises allnecessary voltage power transformation, power regularization (e.g. UPS),power distribution, and cooling subsystems. This configuration generallyallows the simultaneous optimization of the power capacity density andhosting flexibility at very large scales.

A data center module in accordance with the principles of the presentinvention generally comprises an outer envelop and a plurality oflevels, the plurality of levels being superimposed one over the otherand comprising a bottom level and at least two upper levels, the atleast two upper levels comprising a plurality of computing machines, theplurality of levels being in fluid communication thereby allowingdownward and upward movements of air within the module. The modulecomprises an air handling unit, the air handling unit being in fluidcommunication with the top of the at least two upper levels, whereineach of the plurality of levels is partitioned into a first area and asecond area; the first areas of the plurality of levels are in fluidcommunication to allow downward movements of air within the module, andwherein the second areas of the plurality of levels are in fluidcommunication to allow upward movements of air within the module; thefirst area and the second area of the bottom level are in fluidcommunication to allow air moving downwardly into the first area totransfer upwardly into the second area; the computing machines arelocated in one of said first area or said second area of each of the atleast two upper levels; the computing machines are arranged in at leastone row, and wherein the at least one row defines at least two aisles;the at least two aisles comprise at least one cold aisle located on oneside of the at least one row of computing machines, the at least onecold aisle carrying cooling air toward the computing machines, andwherein the at least two aisles comprise at least one hot aisle locatedon the other side of the at least one row of computing machines, the hotaisle carrying warmed cooling air flowing out of the computing machines;the at least one hot aisles have non decreasing cross-section whenflowing from one level to the next.

In typical yet non-limitative embodiments, the data center module isconfigured to be prefabricated and be deployed in clusters of otheridentical (at least externally) data center modules juxtaposed bothside-by-side and back-to-back without interleaving between adjacentmodules.

In typical yet non-limitative embodiments, the data center module has a30-feet by 40-feet footprint, e.g. the equivalent of three 40-feet longshipping containers laid out side-by-side. It can accommodate differentpower density and cooling requirements in various redundancyconfigurations. It combines the advantages of the conventional“brick-and-mortar” data center with those of the container based datacenter, without their respective limitations. Typically using mostlystandardized off-the-shelf electrical and mechanical components, it ismodular and prefabricated to allow fast on-demand deployments, addingcapacity in sync with user needs. It can efficiently host most any typeof computing equipment with any type of power density requirement. Forinstance, power densities of over 30 kilowatts per cabinet are possibleusing air-cooled computer hardware. Cabinets that require chilled-waterfeeds, for instance to support rear-door heat exchangers, are alsopossible, even though rarely required if designed for front-to-back aircirculation. Moreover, low density cabinets can coexist side-by-sidewith high density ones, without creating cooling problems. Formaintenance, large aisles are provided for unconstrained access to boththe front and rear of compute cabinets.

Typically, a module has a ground floor for hosting its power and coolingsubsystems, and several upper floors for hosting its computer cabinets.It is designed to be self-contained and weatherproof. Its maximum powerenvelope is determined by the capacity of its user specified electricalinfrastructure (up to 1.2 megawatts for a typical 30-feet wide unit).Given this infrastructure, the number of upper floors can be adjusted tomatch the power density requirements: less floors for higher density;more floors for lower density. The data center modules are designed toaccommodate any size of air-cooled computer cabinets, as long as aircirculation is front-to-back. The maximum allowable number of cabinetsis a function of the cabinet width and of the number of upper floors.For instance, a 30-feet wide by 40-feet deep unit provides up to two32-feet rows of linear space that can accommodate up to 32 standard size(24-inch wide; 15 per row) cabinets per floor. The average allowablepower dissipation per cabinet is simply determined by dividing the totalpower envelope of the module with its total number of cabinets. Forinstance, a module with a 1.2 megawatts power envelop and threecomputing floors can host up to 96 cabinets, each dissipating 12.5kilowatts on average. With four floors, 128 cabinets could beaccommodated with an average power consumption of 9.4 kilowatts. Thecooling system allows for any mixture of low, medium or high powerdensity cabinets, as long as the total power consumption is below thepower envelope of the module.

Herein, low power density typically refers to 5 kilowatts or less percabinet, medium density typically refers to between 5 and 15 kilowattsper cabinet, and high density typically refers to more than 15 kilowattsper cabinet. However, such ranges are likely to change over time.

In accordance with the principles of the present invention, though eachdata center module is mostly autonomous, it is configured to be deployedaround a central facility responsible for providing reliable low ormedium voltage power feeds that can efficiently be carried overdistances of several hundreds of feet to modules, in a cost-effectiveand energy efficient way.

Herein, low voltage is typically defined as below 1 kilovolt, whilemedium voltage is typically between 1 and 30 kilovolts. The centralfacility typically includes the usual medium voltage power generatorsand transfer switch-gears that provide backup energy in case of gridfailure. It can also include any high-to-medium voltage transformationgear that is necessary if the utility company energizes the centralfacility with a high voltage power line. Herein, high voltage typicallyrefers to above 30 kilovolts.

The central facility typically further includes high efficiency modularchilled-water production subsystems, optimized for the local climateusing water towers or any other water-side economizer mechanisms. Therational for centralizing the chilled-water service revolves around thefollowing three motivations. First, on a yearly basis, it is expectedthat most of the cooling necessary for a module can be realized using anair-side economizer cycle based on outside fresh air. Thus, there is noneed for providing necessarily undersubscribed and inefficient localchilled-water production capacity. The air-side economizer cycle isbuilt into the prefabricated module because, contrary to water, aircannot efficiently be distributed over large distances; it needs to behandled locally. Second, large industrial chillers can be made veryefficient, much more than any other direct exchange (DX) cooling systemsmall enough to fit inside a module. If all cooling cannot be realizedusing an air-side economizer cycle, centralizing the chilled-waterproduction is still an effective way of minimizing the power usageefficiency (PUE) of the data center. Third, if it is practical to reusethe heat generated by the computing machinery for other means, forinstance to heat adjacent buildings during winter, then thechilled-water loop must also be centralized to maximize the energy reuseeffectiveness (ERE) of the data center complex.

Thus, whenever practical, to enable energy reuse, the central facilitycan signal the modules that they should use as much chilled-water asnecessary, by recycling the wasted hot air in a closed-loop,transferring the corresponding energy into the water return of thechilled-water loop. Otherwise, if no more energy reuse is possible, themodules will try to minimize the PUE by using as little chilled-water aspossible, instead favoring free air cooling, breathing outside freshair, circulating this air through computer cabinets and exhausting thewasted hot air to the outside.

Finally, the central facility is also responsible for providing othershared services, for instance points of presence for Internet providers,security check points and biometric access controls, loading docks,meeting rooms, etc.

In typical yet non-limitative embodiments, the central facility isconnected to scalable clusters of data center modules using segregatedpassage ways for power feeds, chilled-water loops, communication networkcables (e.g. fiber-optic cables), and human access. Data center modulesare typically juxtaposed on both sides of a multistory corridorstructure. The ground level corridor generally provides human access tothe power and cooling subsystems, while the upper floor corridors arefor accessing the computing levels. The chilled-water loop is typicallyplaced underground, below the first corridor, while the power feeds arerouted in the false ceiling of the same corridor. All communicationnetwork cables are typically routed in the false ceiling of the secondlevel corridor.

In typical yet non-limitative embodiments, the data center modulecomprises an efficient cooling system combining in a single hybridsystem the efficiency of both air-side and water-side economizers,without multiplying the number of system components. The air-side modeof operation, where the heat dissipated by the computing machinery isrejected into the atmosphere, is preferred when there is no practicalway to reuse this heat, while the water-side mode of operation is usedif the heat can be reused, for example to heat other nearby buildings.The system can efficiently operate partially in both modes (hybrid mode)when only part of the generated heat can be reused in a practical way.

The particular vertical, i.e. multistory, configuration of the datacenter module allows for cost-effective usage of a small number of largemechanical components that both increase efficiency and reliability,contrary to previous modular systems that rely on many more smallercomponents because of either cramped space constraints, or becauseforced-air circulation over long distances is too inefficient.

According to an aspect of the present invention, a deployment method fora data center complex having a plurality of modules operativelyconnected to a central facility is disclosed. Each module preferably hasan air handling unit in fluid communication with the top of at least twoupper levels, each level is generally partitioned into a first and asecond area. The first areas of the levels are in fluid communicationwithin the module. The second areas of the levels are in fluidcommunication within the module. Computing machines are located in onefirst or second area of two upper levels and arranged in at least onerow defining two aisles, one cold aisle located on one side the rowcarrying cooling air toward the computing machines and one hot aislelocated on the other side of the row carrying warmed cooling air flowingout of the computing machines. The method comprises the steps of:

-   -   constructing the central facility for housing the main power        infrastructures shared by the modules;    -   installing medium or high voltage power feeds from a utility        company with adequate voltage transformation, switch gears and        protection systems in the central facility;    -   building foundations for supporting modules;    -   installing a module on the foundations;    -   operatively connecting the module to the central facility;    -   installing and operatively connecting subsequent modules until        the data center complex has the desired capacity.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the foundation to alsosupport access corridors and passageways. According to another aspect ofthe present invention the deployment method may have modules juxtaposedside by side and/or juxtaposed back to back.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the module have amodule specific air side mode of operation and a centralized watersidemode of operation and able to operate in a hybrid mode of operationcombining the use of the air side and water side mode or operations.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein each module has atleast four levels, the lower housing power and cooling subsystemcomponents.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the each module is atleast 30 feet wide by 40 feet deep.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the hot and cold airflow upwardly in the hot and cold air aisles of the module.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the hot and cold aislesare fluidly connected through the computer machine at each of the upperlevels.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the uninterruptiblepower supply (UPS) are located in the lowest level of the modules.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein at least some accesscorridors and passageways are shared by a plurality of modules.

According to an aspect of the present invention the deployment methodmay also be applicable for a data center wherein the air of the airhandling unit in the modules flows downwardly from upper to lowerlevels.

Other and further aspects and advantages of the present invention willbe obvious upon an understanding of the illustrative embodiments aboutto be described or will be indicated in the appended claims, and variousadvantages not referred to herein will occur to one skilled in the artupon employment of the invention in practice. The features of thepresent invention which are believed to be novel are set forth withparticularity in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the inventionwill become more readily apparent from the following description,reference being made to the accompanying drawings in which:

FIG. 1 a presents a perspective view of an embodiment of a large scaledata center complex with a central facility building and 2 clusters of32 prefabricated data center modules each, connected by a grid ofcorridors, in accordance with the principles of the present invention.

FIG. 1 b presents a perspective view of an embodiment of a centralfacility building with three data center modules, but with a corridorsection and concrete slab ready for appending 5 additional prefabricateddata center modules.

FIG. 2 is a perspective view of an embodiment of a four-storyprefabricated data center module in accordance with the principles ofthe present invention, the module comprising a ground floor for powerand cooling subsystems, and three upper floors for computing machinery.

FIG. 3 is a plan view projection of the first upper floor of theprefabricated data center module of FIG. 2, where computing equipments(e.g. servers) are located.

FIG. 4 is a plan view projection of the ground floor of theprefabricated data center module of FIG. 2, where the power and coolingsubsystems are located.

FIG. 5 is an elevation side view of the prefabricated data center moduleof FIG. 2 that shows part of the cooling subsystem on the ground floorand the arrangement of computer cabinets on the upper floors.

FIG. 6 is an elevation front view projection of the prefabricated datacenter module of FIG. 2, illustrating its different internal airflowpatterns.

FIG. 7 is a flowchart that illustrates an exemplary method for deployinglarge scale data center module complexes in accordance with theprinciples of the present invention.

FIGS. 8 a and 8 b is a flowchart that illustrates an exemplaryall-season hybrid-loop control method for the cooling system of theprefabricated data center module in accordance with the principles ofthe present invention.

FIG. 9 is a flowchart that illustrates an exemplary closed-loop controlmethod for the cooling system of the prefabricated data center module,in accordance with the principles of the present invention, when theoutside air conditions do not permit the efficient use of an air-sideeconomizer cycle.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Novel prefabricated data center modules and a method of theirlarge-scale deployment will be described hereinafter. Although theinvention is described in terms of specific illustrative embodiments, itis to be understood that the embodiments described herein are by way ofexample only and that the scope of the invention is not intended to belimited thereby.

Referring to FIG. 1 a, a module-based data center complex is shown at 10to be composed of a main facility building 100 surrounded by clusters205 of prefabricated data center modules or units 200. In this case, 2clusters 205 of 32 modules 200 each. The central facility 100 hosts theservices that are shared by the modules 200: low or medium voltage powerfeeds, chilled-water feeds, demineralized water for humidity control,Internet connections, security check points with biometric accesscontrols, washrooms, meeting rooms, etc.

The data center modules 200 are linked to the central facility 100 by agrid of corridors 300 that not only insure secure human access, but alsoserve as passageways to distribute shared services.

The topology of module clusters 205 is not limited to the example shownin FIG. 1 a. In general, clustered modules 200 are juxtaposed on eachside of a main corridor 310, with possible orthogonal secondarycorridors 320, to both minimize the total footprint of clusters 205 andthe distances over which services must be carried. However, any othertopology can be used to accommodate different shapes of land. Tomaximize flexibility, the data center modules 200 are designed to bejuxtaposed side-to-side and back-to-back without wasting any real estateas shown in FIG. 1 a.

The data center modules 200 are multistory to maximize density anddecrease distances. The ground floor is used for mechanical andelectrical subsystems, while the upper floors host the computingmachinery (e.g. servers). The modules 200 are mostly autonomous; theyonly require a power feed and a chilled-water feed provided by thecentral facility 100. They have their own voltage transformers, UPS(s),power distribution, and cooling system. All controls are embedded withineach module 200, but can be monitored and operated from the centralfacility building 100, or remotely through secure network access.

The corridors 310 and 320 that link the main building 100 to the datacenter modules 200 are also multistory. The ground level corridors 312and 322 provide human access to the ground level of the modules 200while the upper floor corridors 314 and 324 are for accessing the uppercomputing levels (see FIG. 1 b). All water feeds are typically carriedunder the ground level corridors 312 and 322, and all power feeds aretypically carried in the false ceiling of the same corridors. In thisway, the effect of a water pipe fracture is minimized. All communicationnetwork feeds are routed in the false ceiling of the above groundcorridors 314 and 324.

In FIG. 1 b, three existing modules 200 are shown connected to thecentral facility building 100 by a corridor section 310 that can accept5 additional modules; 1 on the same side of the corridor 310 as thefirst three modules 200, and 4 on the opposite side. It should be notedthat this corridor 310 is drawn with its walls removed for illustrationpurposes. In reality, it would be closed on both sides with reusableparty walls.

This figure also illustrates the fact that a module-based data centercomplex 10 can be assembled on-demand, one module 200 at a time, afterhaving built a corridor section 310. Not shown are the emergency exitsthat are typically located at the end of corridors 310 and 320.

Referring to FIG. 2, each prefabricated data center module 200 comprisesa ground level 210 for power and cooling subsystems, and several stories230 for computing machinery. In this particular case, three computingstories 230 are shown. Each floor has an access door in the front, onedoor 212 on the ground floor, and one door 232 on each of the upperfloors 230. The ground level door 212 gives access to the module 200power and cooling subsystems, while the upper level doors 232 provideaccess to the computing machinery. They open into the corridorpassageways 312 and 314 of FIG. 1 b.

A module 200 has a weatherproof prefabricated weight bearing exteriorshell or envelop 250 designed to be shared by adjacent modules 200. Inother words, in the present embodiment, a module 200 built adjacent toan existing module 200 will share a wall with the existing module 200,thereby reducing costs and wasted spaces. Still, in other embodiments,each module 200 could have its own exterior envelop 250 without sharingadjacent wall(s).

The corridors 310 and 320 also share this modular structure so that alarge scale data center complex 10 can be rapidly and efficientlyassembled one module 200 at a time.

Still referring to FIG. 2, an air handling unit 270 is located on therooftop 260 of the module 200 to allow an optimized cooling system thatcan benefit from both air-side or water-side economizers, whileeffectively minimizing its real-estate footprint. When climate isfavorable, the cooling system draws outside air through one or moreintake vents 272 and moves this cold air downward to the ground floor210. The cold air is then pushed upwards to cool the computers locatedon the upper floors 230, and the generated hot air is either exhaustedthrough one or more exhaust vents 274 located at the top part of the airhandling unit 270, or recirculated by mixing it with the intake air. Theair handling unit 270 is designed in such a way that exhausted aircannot recirculate through the intake vents 272. In that sense, theexhaust vents 274 of the air handling unit 270 are located higher thanthe intake vents 272 as best shown in FIG. 2. Moreover, the air intakeis recessed from the module's side so that adjacent modules 200 can beattached side-by-side without wasting any interleaving space at theground level, and without any mutual interference. Both intake andexhaust vents 272 and 274 are respectively equipped with motorizeddampers 276 and 278 (see FIG. 6) that can control their effectivecross-sections and, thus, the volume of air per unit of time that canenter and exit the module 200. When climate is unfavorable, or if thereis a possibility of heat reuse, these dampers 276 and 278 are fullyclosed and the cooling system uses the chilled-water loop provided bythe central facility 100 to cool the closed-loop recirculated air.

FIG. 3 gives a plan view projection of the first upper floor 230 of themodule 200. Each upper floor 230 is divided into three rooms or areas bya drywall 231. The first room is a general purpose staging area 234 thatcommunicates with the external access corridor through the entrance door232. The second is an air handling area 236 that links the rooftop airintake 272 to the ground floor 210. The air handling area 236 typicallycomprises one or more fans 237 for pushing the air from the air handlingunit 270 toward the ground floor 210. The third is a computing room 238that comprises, in the present embodiment, two cold-aisles 239 separatedfrom a central hot-aisle 241 by two parallel rows of computer cabinets240. Understandably, in other embodiments, the number of cold-aisle(s),hot-aisle(s) and row(s) of computer cabinets could be different. Forexample, in some embodiments, there could be one cold aisle, one hotaisle, and one row of computer cabinets, and in still other embodiments,there could be three cold-aisles, two hot-aisles and four rows ofcabinets.

Three doors 246 provide access from the staging area 234 to the threeaisles 239 and 241 of the computing room 238, and a fourth door 248 isfor accessing the air handling area 236.

Through grating in the floor (see also FIG. 6), the cold-aisles 239 areconnected from the ground floor 210 to the top floor 230, creating avertical plenum of cold air. The central hot-aisle 241 is connected fromthe first floor 230 to the rooftop air handling unit 270, forminganother vertical plenum. By traversing the compute cabinets 240, fromcold-aisle 239 to hot-aisle 241, the computing machinery can effectivelybe cooled by transferring the generated heat to the airflow.

For a typical 30-feet wide by 40-feet deep module 200, there is room for32 linear feet of cabinets 240 per row, which is enough to host up to 32standardized 24-inch wide cabinets 240 per floor. The last cabinet 242at the end of each row 240, the one nearest to the drywall 231, can beused to accommodate any necessary voltage transformers. Power istypically distributed to the compute cabinets using overhead busbars244.

Wider modules 200 can accommodate more cabinet aisles 240 using the sameprinciple. Similarly, deeper modules 200 can accommodate longer aisleswith more cabinets 240 per row.

The maximal power envelop of a module 200 is determined by two mainlimiting factors: the capacity of its power and cooling subsystems(transformers, UPSs, fans, and coils) and the width of the gratingsection of its cold-aisles 239 on the first floor 230, which determinesthe maximum velocity of the upward air flow. This is a limiting factor,because too much velocity creates turbulence which in turn inducesdifferences in pressure and temperature. It typically needs to be keptunder 5 meters per second (1000-feet per minute) so that the cold-aisle239 can behave as a plenum and, thus, eliminate all possibilities ofnon-uniform cooling. For the typical module 200 of FIG. 3 with its4-feet and a half wide cold-aisles 239, assuming that compute serverscan effectively be cooled using an airflow of 100 CFM per kilowatt ofheat dissipation, at 20-degree Celsius, this translates to a possiblepower envelop of up to 2.4 megawatts for 96 cabinets, or 25 kilowattsper cabinet on average. For a more typical configuration with cabinetsdissipating on average 12 kilowatts, the maximum air velocity drops wellbelow the critical threshold.

As for air velocity in the central hot-aisle 241, it is much lesscritical, because turbulence there will not affect the cooling of thecomputing machinery.

Notably, this configuration enables the electrical subsystems of themodule 200 to be air-cooled with the same system used for cooling thecomputing machinery. No addition components are necessary.

Referring to FIG. 4, the ground floor 210 of the module 200 is alsodivided into three rooms or areas: an entrance hall 214 that can hostfire protection systems, for instance, an electrical room 216 that canhost heat producing electrical components like transformers and UPSs,and a positive pressure intake plenum area 218. The airflow is forceddownward from the above air handling area 236 (see FIGS. 3 and 6) usinga set of variable drive fans 237. It then traverses to the electricalroom 216 through filters 220 and cooling coils 222 before moving upwardthrough the grating floor of the above cold-aisles 239, carrying anyheat dissipated by the electrical components located in the electricalroom 216.

Referring to FIG. 5, the module 200 is divided logically into threevertical parts: a lower part 292 for the electrical and mechanicalcomponents, including cooling coils 222, a multilevel middle part 294for the computing machinery 240, and an upper part 296 for the airhandling unit 270 with its intake vents 272 and exhaust vents 274.

FIG. 6 provides a detailed elevation view of the various airflows insidethe present embodiment of the data center module 200. It indicates thethree vertical parts of the module: lower part 292, middle part 294 andupper part 296; and illustrates that each part can be further subdividedinto a left-hand side 293, and a right-hand side 295. It shows thedifferent system components: filter banks 220, cooling coil sets 222,variable drive fans 237, humidifiers 224, intake vent 272 with dampers276, exhaust vent 274 with dampers 278, computing cabinets 240, upwardvertical hot-aisle 241, upward vertical cold-aisles 239, air mixingdampers 275, downward mixing plenum 236, input plenum 218, cold plenum216, exhaust plenum 277, and optional UPS submodule(s) 217.

Starting from the input plenum 218 that is under a positive pressurecreated by the fans 237, the air first crosses the filters banks 220 andcoils 222. Depending on the mode of operation, this input air can beeither hot or cold. In closed-loop operations, being recirculated fromthe hot-aisle 241 through the exhaust plenum 277, mixing dampers 275,and mixing plenum 236, the air is warm and may need to be cooled by thecoils 222. In hybrid-loop operations, coming mostly from the outsidethrough the intake vent 272, it may be cold enough to not require anycooling, but it may also be too cold. In that case, it is heated usingthe warm air from the exhaust plenum 277, by mixing part of it throughthe mixing dampers 275. Then, whatever warm air from the exhaust plenum277 not used for mixing will naturally exit through the exhaust vents274.

Once the air crosses to the central cold plenum 216, it can rise throughthe grating floors of the cold-aisles 239 and reach the servers in thecomputer cabinets 240. From there, through the computer cabinets 240, itcrosses to the hot-aisle 241, absorbing the heat dissipated by theservers. The rows of cabinets 240 need to form a sealed barrier betweenthe cold-aisles 239 and hot-aisle 241, effectively limiting anyhorizontal air movement to the computer servers inside the cabinets 240.Specifically, lightweight filler panels installed above the cabinets 240serve this purpose. This is another key to efficient air cooling,avoiding any mixture of cold and hot air outside of computer cabinets240. Inside the cabinets 240 themselves, depending on their design, someweather striping materials can also be used to fill smaller holes.

Once in the hot-aisle 241, the air is free to rise through the gratingsto the exhaust plenum 277 where it is either recirculated downwardthrough the mixing dampers 275 and mixing plenum 236, or exhaustedupward through the exhaust vents 274, depending on the modes ofoperation previously described.

Notably, the cooling system of the present embodiment of the module 200can be built from standardized industrial parts readily available andmanufactured in large quantities at low cost. Its global efficiencystems from the use of large capacity and high efficiency fans 237 andcoils 222 that can be made much more efficient than their smallercounterparts usually found in conventional computing room airconditioner (CRAC) units. Moreover, the whole module 200 can beassembled rapidly from manufactured parts using well known and masteredmetal structure building techniques. External party walls and weightbearing structures of modules 200 can be designed so that a new module200 can attach to an existing one. Similarly, corridors 310 and 320 canbe designed in a modular fashion so that new sections can be added withnew modules 200.

Furthermore, each module 200 comprises its own electrical systems,complete with voltage transformations, switch gear protection, and UPS217, possibly in 1n, n+1 or 2n redundant configurations. The ability toregroup all mechanical and electrical systems in a single autonomousmodule 200 is not only cost effective, it is scalable and flexible. Amodule operator can customize the capacity and resiliency of his module200 to satisfy the specific needs of his users, without affecting theoperations of other modules 200. For instance, some modules 200 couldoperate with either no, partial, or full UPS protection, with or withoutredundancy. A priori decisions need not be taken for the whole site, nordo power density zones need to be defined. Decisions can be postponed totime of deployment, one module 200 at a time, or in blocks of multiplemodules 200. Modules 200 can be built on-demand. Upgrades of existingmodules 200 are also feasible without affecting the operations ofothers. For instance, transformers could be upgraded to change from a 1nconfiguration to an n+1 configuration, or a UPS could be added, orsupport extended, if needs evolve over time.

Power distribution to computer cabinets is also flexible. It can rely ondifferent technologies like classical breaker panels, busbars, or in-rowPDU cabinets. Again, the choice need not be taken a priori for the wholesite, but can be postponed to deployment time. The modularity of themodule 200 allows for cost-effective and resilient evolution of the datacenter complex 10 over time.

The problem of cooling the heat dissipation of electrical componentswithin the module 200 is also addressed by placing these componentsinside the cooling system, which is both a cost-effective and energyefficient solution. It then becomes a non-issue. The same is true forthe control systems that include fan drives, valve controls, temperaturesensors, differential pressure sensors, humidity sensors and controls,fire detection and protection, and access controls.

Referring to FIG. 7, the deployment method for a large scale data centercomplex 10 is described by a flowchart. The method bootstraps (at 701)by constructing the central facility building 100 for housing the mainpower and cooling infrastructures that are shared by all modules 200.This initial facility 100 is essentially an empty shell built on aconcrete slab. It has some office space for administration, security,and maintenance staff, but most of its footprint is typically of lowcost warehouse type. It must generally be sized according to theexpected maximum power capacity of the whole data center complex 10.Then, the corresponding medium or high voltage power feeds from theutility company must be installed with adequate voltage transformation,switch gears, and protection systems. If possible, this step shall bephased to minimize initial investments. The important thing is to haveenough switch gear to make sure that additional power capacity can beadded without having to interrupt services to existing modules 200.Backup generators and chillers modules should generally be installed oneby one, as user needs evolve, maximizing ROI. Building modules 200requires a concrete slab with strong foundations because of the weightof the computing machinery. As building these foundations may take asomewhat long lead time, especially for locations where the groundfreezes during winter, it may be wise to anticipate user needs and buildthem well in advance for at least several (e.g. 4) modules 200,including access corridors and passageways 310 and 320. Obviously, thisnumber can be increased if rapidly changing user needs are expected. Thelast step of this initial work is to build and setup the first module200 to address the initial user needs. Again, if these needs areinitially greater, the number of initial modules 200 should be augmentedaccordingly.

Afterward, user needs are constantly assessed (at 702) and if no longerfulfilled, a new module 200 is ordered, factory built and assembled onexisting foundations (at 705). If no foundations are available (at 703),or if not enough of them are currently available to address the expectedshort term needs, then new foundations are built in increments oftypically 4 or more (at 704). If medium voltage power or coolingcapacity is short in the central facility 100 (at 706), but space andenergy is still available (at 707), then new power and/or coolingmodules are added to the main building 100 (at 708). Otherwise, if powerand cooling capacity for the new modules 200 is short and space orenergy is exhausted, then the site has reached its capacity and a newdata center complex 10 must be built on a new site.

Referring to FIG. 8 a, the hybrid-loop control method 800 for cooling amodule 200 is described with the help of a flowchart. This method 800applies independently for each of the two cooling subsystems in themodule 200. The method starts (at 801) by initially fully opening theintake and exhaust dampers 276 and 278, and fully closing the mixingdampers 275. The chilled-water valve is also initially closed so that nowater is flowing through the coils 222. Finally, the humidifiers 224 arealso initially shutoff.

Then, the method 800 enters a loop where outside air conditions arefirst evaluated. If temperature or humidity are out of limits (“yes”branch at 802), then the system may no longer operate in hybrid-loop andis automatically switched to closed-loop operation (see 900 of FIG. 9).Indeed, when the outside temperature nears the set-point temperature forthe cold air plenum, the system can no longer operate in hybrid-loop inany practical way, so it reverts to closed loop operations. The decisioncan be implemented using either the outside dry bulb temperature or themore precise air enthalpy. If the outside conditions are favorable (“no”branch at 802), then the process continues by measuring the differentialpressure on all floors 230, between the cold and hot aisles 239 and 241,for all cabinet rows 240. The lowest measurement is kept and used toadjust the fan speed (at 805) if the pressure is determined to be out oflimits (“yes” branch at 804). The acceptable range of differentialpressure is between two small positive values. In the case where thecold-aisles 239 are maintained at temperatures below 20 degrees Celsius,the lower end of this range should be approximately zero; if thecold-aisle 239 is operated at higher temperature, it may need to besomewhat above zero to maintain a more aggressive minimum differentialpressure. The fan speed adjustment method uses standard controlalgorithms for this purpose.

The next step is to regulate the temperature of the cold-aisles 239 ifit is outside of the preset limits (at 806). The temperature is measuredat the output of the cooling subsystem in the central cold air plenum216, below the first computing level 230. Four variables can becontrolled to achieve temperature regulation: the flow of water in thecoils 222, and the flow of air in the intake, exhaust, and mixingdampers 276, 278 and 275 respectively (at 807).

Referring to FIG. 8 b, the method performed at 807 for adjusting thedampers and water flow is illustrated with a flowchart. When the currentcold-aisle 239 temperature is too cold (“too cold” branch at 810), themethod uses a strategy that prioritize the variables in the followingorder: water flow, mixing airflow, exhaust airflow, and intake airflow.If water is currently flowing, but not being reused by the centralfacility 100 (“yes” branch at 819), then its flow is decreased (820) tomaximize the use of the air-side economizer cycle (which is the generalobjective of the hybrid-loop operation). Otherwise (“no” branch at 819),either no water is flowing, in which case flow cannot be reduced, orwater is flowing, but needed by the central facility 100 for usefulenergy reuse. At this point, some warm air from the exhaust plenum 277must be recirculated to further preheat the air in the mixing plenum236. If the mixing dampers 275 are not yet fully opened (“no” branch at821), then it is opened some more to increase air mixing (at 822). Inthis way, more of the warm air in the exhaust plenum 277 is mixed withthe external cold air to raise the air temperature of the input plenum218. On the contrary, if the mixing dampers 275 are already fully opened(“yes” branch at 821), then it is necessary to act on the exhaustdampers 278 by decreasing the flow of air that can exit the module 200(at 824). In this way, more of the exhaust plenum air can mix with theoutside air to raise the temperature in the input plenum 218. In theextreme case, the exhaust dampers 278 are fully closed (“yes” branch at823) and all of the warm hot-aisle 241 air is recirculated. When thishappens, there is a possibility that some of this warm air underpressure will exit through the intake vent 272 instead of being suckeddownward in the mixing plenum 236, so the intake damper 276cross-section needs to be decreased (at 825) to create a restrictionthat will force all the mixed air to flow downwards. It is not possiblethat the intake dampers 276 fully close unless no heat is dissipated bythe computing machinery.

If the cold-aisle 239 temperature is too warm (“too warm” branch at810), then the strategy is to prioritize the control variables in thereverse order: intake airflow, exhaust airflow, mixing airflow, andwater flow, assuming that water is currently not being reused by thecentral facility (“no” branch at 811). If the intake dampers 276 are notfully opened (“no” branch at 812), then they should be opened some moreto increased the intake airflow (at 813) and allow the possibility formore cold air to enter. Otherwise, they are already fully opened (“yes”branch at 812) and it is the exhaust dampers 278 that need to be openedto allow increased air exhaust (at 815) and, thus, increased airexchange with the outside. Otherwise, both intake and exhaust dampers276 and 278 are fully opened, and it is the mixer dampers 275 that needto be closed some more if it is not already fully closed (“no” branch at816), to decrease air mixing (at 817) and reduce the warming of theoutside air. Otherwise, if the mixing dampers 275 are fully opened(“yes” branch at 816), or if the water is currently being reused by thecentral facility 100 (“yes” branch at 811), then the coils 222 need toabsorb more heat by increasing their water flow (at 818).

Back to FIG. 8 a, the next step is adjusting the humidifier output (at809) if the relative humidity in the cold air plenum 216 is out oflimits (“yes” branch at 808) for normal operations of the computerservers, as specified by the computer manufacturers. The method formaking this adjustment again uses standard algorithms. After this step,the process starts over by checking repeatedly outside air conditions,differential pressure, cold air plenum temperature, and humidity, and bymaking adjustments, whenever necessary.

The humidifiers increase relative humidity, essentially when the outsideair temperature is very cold, and thus too dry once it has been warmedto its set-point temperature. For this purpose, the humidifiers 224vaporize demineralized water using an efficient adiabatic mechanism.During the summer time, the relative humidity inside the module 200 canalso become too high if the outside air is too humid. In those cases,however, the system will tend to switch to closed-loop operations,because the air enthalpy probably makes the air-side economizer cyclecounterproductive. In any case, the excessive humidity will be removedby the cooling coils 222 through condensation.

Referring to FIG. 9, the closed-loop control method 900 for cooling themodule 200 is described with the help of a flowchart. The closed-loopmethod 900 is similar to the hybrid-loop one, but simpler because thetemperature regulation has a single variable to work with: the flow ofchilled-water in the coils 222. The method 900 starts by fully closingthe intake and exhaust dampers 276 and 278, and fully opening the mixingdampers 275 so that all the air in the exhaust plenum 277 isrecirculating into the input plenum 218. The chilled-water valve is alsoinitially closed so that no water is flowing through the coils 222, andthe humidifiers 224 are shutoff.

Then, the method enters a loop where outside air conditions are firstevaluated. If temperature and humidity are within limits (“yes” branchat 902), then the system can switch back to hybrid-loop operations usingthe air-side economizer cycle. It should be noted here that the outsidecondition limits for switching from closed-loop to hybrid-loop are notnecessarily the same as the one for switching from hybrid-loop toclosed-loop. Some hysteresis should be used so that the system does notoscillate between the two modes of operation. If outside conditions areunfavorable (“no” branch at 902), then the method continues by measuringthe differential pressure on all floors, between the cold and hot aisles239 and 241, on both sides of the cabinet rows 240. The lowestmeasurement is kept and used to adjust the fan speed (at 904) if thedifferential pressure is determined to be out of limits (“yes” branch at903). The acceptable range of differential pressure is between two smallpositive values. In the case where the cold-aisle 239 is maintained attemperatures below 20 degrees Celsius, the lower end of this rangeshould be approximately zero; if the cold-aisles 239 are operated athigher temperature, it may need to be somewhat above zero to maintain amore aggressive minimum differential pressure. The speed adjustmentmethod uses standard control algorithms for this purpose.

The next step is to regulate the temperature of the cold-aisle 239 bycontrolling the flow of water in the coils 222. The temperature ismeasured at the output of the cooling subsystem in the cold air centralplenum 216. When the current temperature is out of limits (“yes” branchat 905), the method simply adjusts the water flow (at 906) in the coils222 using standard control algorithms for this purpose.

The final step is adjusting the humidifier output (at 908) if therelative humidity in the cold air plenum 216 is out of limit (“yes”branch at 907) for normal operations of servers, as specified by thecomputer manufacturers. The method for making this adjustment again usesstandard control algorithms. After this step, the process starts over bychecking repeatedly outside air conditions, differential pressure,temperature, and humidity, and by making adjustments, whenevernecessary.

While illustrative and presently preferred embodiments of the inventionhave been described in detail hereinabove, it is to be understood thatthe inventive concepts may be otherwise variously embodied and employedand that the appended claims are intended to be construed to includesuch variations except insofar as limited by the prior art.

1) A deployment method for a data center complex having a plurality ofmodules operatively connected to a central facility, each module havingan air handling unit in fluid communication with the top of at least twoupper levels, each level is partitioned into a first and a second area,the first areas of the levels are in fluid communication within themodule, the second areas of the levels are in fluid communication withinthe module, computing machines are located in one first or second areaof two upper levels and arranged in at least one row defining twoaisles, one cold aisle located on one side the row carrying cooling airtoward the computing machines and one hot aisle located on the otherside of the row carrying warmed cooling air flowing out of the computingmachines, the method comprising the steps of: a) constructing thecentral facility for housing the main power infrastructures shared bythe modules; b) installing medium or high voltage power feeds from autility company with adequate voltage transformation, switch gears andprotection systems in the central facility; c) building foundations forsupporting modules; d) installing a module on the foundations; e)operatively connecting the module to the central facility; f) installingand operatively connecting subsequent modules until the data centercomplex has the desired capacity. 2) A deployment method for a datacenter complex of claim 1 wherein the foundation also supports accesscorridors and passageways. 3) A deployment method for a data centercomplex of claim 1 wherein the modules are juxtaposed side by side. 4) Adeployment method for a data center complex of claim 1 wherein themodules are juxtaposed back to back. 5) A deployment method for a datacenter complex of claim 1 wherein the modules are juxtaposed side byside and back to back. 6) A deployment method for a data center complexof claim 1 wherein each module have a module specific air side mode ofoperation and a centralized waterside mode of operation and able tooperate in a hybrid mode of operation combining the use of the air sideand water side mode or operations. 7) A deployment method for a datacenter complex of claim 1, wherein each module has at least four levels,the lower housing power and cooling subsystem components. 8) Adeployment method for a data center complex of claim 1, wherein eachmodule is at least 30 feet wide by 40 feet deep. 9) A deployment methodfor a data center complex of claim 1, wherein the hot and cold air flowupwardly in the hot and cold air aisles of the module. 10) A deploymentmethod for a data center complex of claim 1, wherein the hot and coldaisles are fluidly connected through the computer machine at each of theupper levels. 11) A deployment method for a data center complex of claim1, wherein the uninterruptible power supply (UPS) are located in thelowest level of the modules. 12) A deployment method for a data centercomplex of claim 2, wherein at least some access corridors andpassageways are shared by a plurality of modules. 13) A deploymentmethod for a data center complex of claim 1 wherein air of the airhandling unit in the modules flows downwardly from upper to lowerlevels.